Popular DataScience Tags

Finding popular tags in Data Science: Analyzing people's interests in Data Science¶

Introduction¶

In this project, we're going to find the popular tags that people use to ask questions related to Data Science. Tags here means different programming languages, frameworks, and packages used in Data Science. Through our analysis we're trying to see interest of the majority of people in the community. This helps us to get an idea of where the Data Science landscape is moving towards.

To analyze, we're going to take a look at data from Stack Exchange public database. Stack Exchange is the popular site for questions and answers related to the field. You can check the link here to explore more yourself. So, for now, we're following steps below for our purpose:

• find the appropriate features for data collection from the tables
• apply the SQL query to extract the data and download it
• find the criteria or proxy to determine the popularity of technologies
• find the top 20 tags(technologies)

Summary of Results¶

After analyzing the data, we got an idea of people using python more often than R in the data science community. From the Stack Exchange data, we can be clear that more people are using python to solve their Data Science related problems. The use of python libraries as tags to ask questions in the Data Science platform supports this fact. But compared to 2019, the year 2020 has lower numbers for all the top 20 tags used. We can conclude, based on data, python is dominating the field of Deep Learning and AI (Artificial Intelligence) as quarter of the tags used in the top 20 tags are of deep learning.

For details, please follow full analysis.

Note: If you're looking in Github, graphs drawing using plotly don't appear as github doesn't support it (at time of writing). Either you have to run locally or in google-colab to see the graphs plotted using plotly. So, for the reader's convenience, I've used seaborn and matplotlib to plot the same graphs again. The reason to use plotly as it gives more flexibility while interacting with the graphs using effects like zooming and many more without extra coding.

Exploring the public StackExchange database¶

You can check the link for the Stack Exchange public database where you can try your own query to get answer related to data.

After running some query, found Posts tables has some valuable features to answer the question for most popular topics. For example,Posts table has two columns Tags & ViewCount serves better for our purpose of analysis.

The Posts table has following columns(that we're considering):

• Id: An identification number for the post.

• PostTypeId: An identification number for the type of post.

• CreationDate: The date and time of creation of the post.

• Score: The post's score.

• ViewCount: How many times the post was viewed.
• Tags: What tags were used.
• AnswerCount: How many answers the question got (only applicable to question posts).
• FavoriteCount: How many times the question was favored (only applicable to question posts).

Using Following SQL command to extract the required data of 2019 & 2020 from the database.

SELECT Id,PostTypeId, CreationDate,Score,ViewCount, Tags,AnswerCount, FavoriteCount FROM posts WHERE CreationDate >= '01/01/2019 00:00:00';

After running the above query in the Stack Exchange public database, click the download link for the csv file.

Now, let's load the csv file into dataFrame and start the analysis.

In [2]:

# load standard libraries for data manipulation
import pandas as pd
import numpy as np

# libraries for visaulization
import matplotlib.pyplot as plt
import seaborn as sns

sns.set()  # setting parameters for seaborn

%matplotlib inline

In [140]:

# let's read the data in csv file, load to pandas dataFrame
df = pd.read_csv('data/popularDS.csv',parse_dates=['CreationDate'])

# data info
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 24997 entries, 0 to 24996
Data columns (total 8 columns):
Id               24997 non-null int64
PostTypeId       24997 non-null int64
CreationDate     24997 non-null datetime64[ns]
Score            24997 non-null int64
ViewCount        12755 non-null float64
Tags             12755 non-null object
AnswerCount      12755 non-null float64
FavoriteCount    2043 non-null float64
dtypes: datetime64[ns](1), float64(3), int64(3), object(1)
memory usage: 1.5+ MB

In [12]:

# let's check the number of missing values in each features(columns)
df.isnull().sum()[df.isnull().sum() >0]

Out[12]:

ViewCount        12242
Tags             12242
AnswerCount      12242
FavoriteCount    22954
dtype: int64

As we can see, there are same 12242 missing values for all the features except FavoriteCount which has 22954.

In [13]:

# let's explore the first 10 rows of our data
df.head(10)

Out[13]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
0	46451	2	2019-03-01 08:56:23	8	NaN	NaN	NaN	NaN
1	46452	1	2019-03-01 09:01:14	0	131.0	<lstm><loss-function><epochs>	1.0	NaN
2	46453	1	2019-03-01 09:02:19	2	96.0	<deep-learning><cnn>	0.0	1.0
3	46454	1	2019-03-01 09:08:53	0	653.0	<time-series><clustering><unsupervised-learnin...	2.0	NaN
4	46455	2	2019-03-01 09:21:03	0	NaN	NaN	NaN	NaN
5	46456	1	2019-03-01 09:22:37	1	1174.0	<machine-learning><python><dataset><training><...	1.0	0.0
6	46457	1	2019-03-01 09:59:36	2	58.0	<lstm><code>	1.0	1.0
7	46459	1	2019-03-01 10:27:30	1	26.0	<lstm>	0.0	2.0
8	46460	2	2019-03-01 10:45:16	0	NaN	NaN	NaN	NaN
9	46462	2	2019-03-01 12:01:22	2	NaN	NaN	NaN	NaN

In [14]:

# let's see the stats for the numerical values
df.describe()

Out[14]:

	Id	PostTypeId	Score	ViewCount	AnswerCount	FavoriteCount
count	24997.000000	24997.000000	24997.000000	12755.000000	12755.000000	2043.000000
mean	61020.114574	1.509821	1.014562	278.892434	0.848373	1.268722
std	9919.103822	0.569215	1.728197	1536.342332	0.865539	1.138213
min	43360.000000	1.000000	-4.000000	2.000000	0.000000	0.000000
25%	52877.000000	1.000000	0.000000	22.000000	0.000000	1.000000
50%	61694.000000	1.000000	1.000000	42.000000	1.000000	1.000000
75%	69105.000000	2.000000	1.000000	125.000000	1.000000	1.000000
max	77578.000000	6.000000	45.000000	88393.000000	10.000000	16.000000

Tags column is the only categorical feature in our data which has multiple values surrounded by arrow-brackets. Next step is the data cleaning in our steps:

Data Cleaning¶

We can fill the numerical missing values with median or mean or appropriate value as required(whichever better suits for particular features and case). While for catergorical columns we can fill the missing values with most frequent or with other appropriate values.

In [3]:

# cleaning the Tags columns by removing the arrow-brackets & 
# separating words by comma with last character stripped
df['Tags'] = df['Tags'].str.replace('<','').str.replace('>',',').str[:-1]

In [16]:

df.head()

Out[16]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
0	46451	2	2019-03-01 08:56:23	8	NaN	NaN	NaN	NaN
1	46452	1	2019-03-01 09:01:14	0	131.0	lstm,loss-function,epochs	1.0	NaN
2	46453	1	2019-03-01 09:02:19	2	96.0	deep-learning,cnn	0.0	1.0
3	46454	1	2019-03-01 09:08:53	0	653.0	time-series,clustering,unsupervised-learning,a...	2.0	NaN
4	46455	2	2019-03-01 09:21:03	0	NaN	NaN	NaN	NaN

In [4]:

# filling the missing values for numerical features

# list of columns for filling
fill_list = ['ViewCount','AnswerCount','FavoriteCount']

# loop over the list
for l in fill_list:
    # filling the missing values with 0
    df[l].fillna(0,inplace=True)
    # converting the data type to int
    df[l] = df[l].astype(int)
    
# info after changes
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 24997 entries, 0 to 24996
Data columns (total 8 columns):
Id               24997 non-null int64
PostTypeId       24997 non-null int64
CreationDate     24997 non-null datetime64[ns]
Score            24997 non-null int64
ViewCount        24997 non-null int32
Tags             12755 non-null object
AnswerCount      24997 non-null int32
FavoriteCount    24997 non-null int32
dtypes: datetime64[ns](1), int32(3), int64(3), object(1)
memory usage: 1.2+ MB

You can see except Tags there are no missing values in the columns. We'll handle Tag column separately later.

In [21]:

# let's look at random 5 rows
df.sample(5)

Out[21]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
694	60730	1	2019-09-25 08:06:58	0	9	machine-learning,neural-network,deep-learning,...	0	0
14018	55262	2	2019-07-08 08:40:27	6	0	NaN	0	0
2559	68494	1	2020-02-22 00:05:10	2	19	machine-learning,recurrent-neural-net	0	0
23242	60437	1	2019-09-19 08:04:11	0	101	scikit-learn,preprocessing,pickle	1	0
16836	55400	1	2019-07-10 05:24:25	1	127	lstm	1	0

In [22]:

# save the clean data into csv file
df.to_csv('clean_popularDS.csv',index=False)

Loading the clean data¶

In [3]:

df_clean = pd.read_csv('data/clean_popularDS.csv')
df_clean.head()

Out[3]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
0	46451	2	2019-03-01 08:56:23	8	0	NaN	0	0
1	46452	1	2019-03-01 09:01:14	0	131	lstm,loss-function,epochs	1	0
2	46453	1	2019-03-01 09:02:19	2	96	deep-learning,cnn	0	1
3	46454	1	2019-03-01 09:08:53	0	653	time-series,clustering,unsupervised-learning,a...	2	0
4	46455	2	2019-03-01 09:21:03	0	0	NaN	0	0

Criteria for most popular tags¶

Now, to determine most popular tags we're using two proxies: Number of Tags used plus ViewCount for the post(question).

In [4]:

# function to return count of values
def popular_content(tdf):
    """
    Function to return the count of values from given string
    
    Args:
        tdf: dataFrame
        
    Returns: 
        two dataFrame containing most used tags and most viewed tags
    """
    
    # splitting the Tags values of string to generate list in the series
    tag_series = tdf['Tags'].apply(lambda x: x.split(','))
    
    #creating empty dict
    tag_dict = dict()
    view_dict = dict()
    
    # looping over Tags series
    for index,item in tag_series.items():
        # looping over the items in the list
        for val in item:
            #checking if the value exists in the dict
            if val not in tag_dict:
                tag_dict[val] =1
                
                # catching index out-of-bounds error
                try:
                    view_dict[val] = tdf['ViewCount'].iloc[index]
                except:
                    pass
                
            # if already value already exists increase the count
            else:
                tag_dict[val] +=1
                try:
                    view_dict[val] += tdf['ViewCount'].iloc[index]
                except:
                    pass
            
    # converting the dict into dataFrame
    used_df = pd.DataFrame.from_dict(tag_dict,orient='index',
                                    columns=['Count'])
    
    # most viewed tag
    viewed_df = pd.DataFrame.from_dict(view_dict,orient='index',
                                      columns=['ViewCount'])
    
    return used_df,viewed_df

In [5]:

# for this purpose we're taking dataFrame without null Tags values &
# resetting the index for caculation purpose
questions_df = df_clean[df_clean['Tags'].notnull()].reset_index(drop=True)

questions_df.head()

Out[5]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
0	46452	1	2019-03-01 09:01:14	0	131	lstm,loss-function,epochs	1	0
1	46453	1	2019-03-01 09:02:19	2	96	deep-learning,cnn	0	1
2	46454	1	2019-03-01 09:08:53	0	653	time-series,clustering,unsupervised-learning,a...	2	0
3	46456	1	2019-03-01 09:22:37	1	1174	machine-learning,python,dataset,training,objec...	1	0
4	46457	1	2019-03-01 09:59:36	2	58	lstm,code	1	1

In [6]:

# calling the function popular_content
most_used_df, most_viewed_df = popular_content(questions_df)

In [29]:

# checking most used tags
most_used_df.head(10)

Out[29]:

	Count
lstm	544
loss-function	241
epochs	16
deep-learning	1705
cnn	667
time-series	646
clustering	422
unsupervised-learning	173
anomaly-detection	130
machine-learning	3785

In [63]:

# checking most viewed tags
most_viewed_df.head(10)

Out[63]:

	ViewCount
lstm	157203
loss-function	51565
epochs	23117
deep-learning	496582
cnn	166664
time-series	142171
clustering	75597
unsupervised-learning	25396
anomaly-detection	21957
machine-learning	865175

In [7]:

# let's merge the datasets to see the scatter plot between used tags
# vs viewed tags
combined_df = pd.merge(most_used_df,most_viewed_df,how='left',
                      left_index=True,right_index=True)
combined_df.sort_values(by='Count',ascending=False,inplace=True)
combined_df.head()

Out[7]:

	Count	ViewCount
machine-learning	3785	865175
python	2594	1300942
deep-learning	1705	496582
neural-network	1488	416241
keras	1291	595612

In [8]:

# filtering tags with count more than 100 over the period
combined_100_df = combined_df[combined_df['Count'] > 100]
# top 20 tags based on used count
top_20_tags = combined_100_df[:20]
# bottom 20 after filter based on used count
bottom_20_tags = combined_100_df[-20:]

In [96]:

# scatter plot using plotly
import plotly.express as px

tags_index = top_20_tags.index.values

fig = px.scatter(top_20_tags, x="Count", y="ViewCount", text=tags_index, log_x=True, size_max=100)
fig.update_traces(textposition='top center')
fig.update_layout(title_text='Top 20 tags used in Data Science posts', title_x=0.5)
fig.show()

In [10]:

# above graph implemented using matplotlib

x = top_20_tags[:10]['Count']
y= top_20_tags[:10]['ViewCount']
names = top_20_tags[:10].index.values

fig, ax = plt.subplots(figsize=(8,6))
ax.scatter(x,y)

for i, txt in enumerate(names):
    ax.annotate(txt,(x[i],y[i]),xytext=(x[i]+100,y[i]+10),
               arrowprops = dict(facecolor='green',arrowstyle='fancy'))
plt.title('Top 10 tags in Data Science posts')
plt.xlabel('Used Count')
plt.ylabel('View Count')
plt.show()

From above figure, we can see python is at the top considering number of view count. While machine-learning tag is at the top considering number of used count. This gives us an idea of more questions were asked using machine learning tag while more questions were viewed related to python tag.

In [97]:

# scatter plotting using plotly
tags_index = bottom_20_tags.index.values

fig = px.scatter(bottom_20_tags, x="Count", y="ViewCount", text=tags_index, log_x=True, size_max=100)
fig.update_traces(textposition='top center')
fig.update_layout(title_text='Bottom 20 tags used in Data Science posts', title_x=0.5)
fig.show()

In [133]:

# above graph implemented using matplotlib

x = bottom_20_tags['Count']
y= bottom_20_tags['ViewCount']
names = bottom_20_tags.index.values

fig, ax = plt.subplots(figsize=(8,6))
ax.scatter(x,y)

for i, txt in enumerate(names):
    ax.annotate(txt,(x[i],y[i]),xytext=(x[i]+1,y[i]-1),
               arrowprops = dict(facecolor='green',arrowstyle='fancy'))
plt.title('Bottom 20 tags used in Data Science posts')
plt.xlabel('Count')
plt.ylabel('ViewCount')
plt.show()

You can notice different machine learning terminology like overfitting, class-imbalance, backpropagation and so on find place in filtered bottom 20 tags. Tag bert(technique for Natural language processing) is at top related to view count whereas tags categorical-data and anomaly-detection placed top related to used count.

Top 20 most used and most viewed Tags¶

Let's analyze the tags most used(for asking questions) and most viewed(people looked for) tags separately.

So, to get the top 20 for both of the tags we've to:

• format dataFrame's into descending order
• use head to take first 20

In [7]:

# top 20 most used tags 
top_20_used_tags = most_used_df.sort_values(by=['Count'],
                                            ascending=False).head(20)

# top 20 most viewed tags
top_20_viewed_tags = most_viewed_df.sort_values(by=['ViewCount'],
                                            ascending=False).head(20)

Visualization for the top 20 tags¶

Here, we're using line graph to show the top 20 tags in both used and viewed tags. After that, we're using the pie-chart to see what portion each tags correspond with respect to total. We're using plotly library here.

In [48]:

import plotly.express as px

labels = top_20_used_tags.index.values

fig = px.bar(top_20_used_tags, x=labels,
             y="Count",
             title="Top 20 tags used in Data Science",
            color=labels)
fig.update_layout(showlegend=False)
fig.show()

Machine Learning is at the top, 3785 times people have tagged it on the posts. Following its footsteps we can see python, 2594 times tagged in the post. We can see people have used r 398 times on their posts, bottom forth among the top 20 tags.

In [43]:

# line graph using seaborn as github doesn't display plotly graphs
labels = top_20_used_tags.index.values

fig,ax = plt.subplots(figsize=(15,6))
ax = sns.barplot(x=labels,y='Count',data=top_20_used_tags)
plt.xticks(rotation=65,fontsize=15)
plt.yticks(fontsize=15)
plt.title("Top 20 tags used in Data Science",fontsize=20)
plt.show()

In [49]:

labels = top_20_viewed_tags.index.values

fig = px.bar(top_20_viewed_tags, x=labels,
             y="ViewCount",
             title="Top 20 tags viewed in Data Science",
            color=labels)
fig.update_layout(showlegend=False)
fig.show()

In the case of view count, post with python tag has the viewer of 1.3M while r tag has viewer of 88.9K (thousand) times residing in the top and bottom of the top 20 list.

In [44]:

# line graph using seaborn as github doesn't display plotly graphs
labels = top_20_viewed_tags.index.values

fig,ax = plt.subplots(figsize=(15,6))
ax = sns.barplot(x=labels,y='ViewCount',data=top_20_viewed_tags)
plt.xticks(rotation=65,fontsize=15)
plt.yticks(fontsize=15)
plt.title("Top 20 tags viewed in Data Science",fontsize=20)
plt.show()

In [44]:

import plotly.graph_objects as go

labels = top_20_used_tags.index.values
values = top_20_used_tags['Count']

fig = go.Figure(data=[go.Pie(labels=labels, values=values, 
                             textinfo='label+percent',
                             insidetextorientation='radial'
                            )])

fig.update_layout(showlegend=False,title='Top 20 used Data Science Tags')
fig.show()

In [56]:

# same for the pie-charts

#Pie chart
labels = top_20_used_tags.index.values
sizes = top_20_used_tags['Count']

# only "explode" the first slice (i.e. value with highest count)
explode = (0.1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0)

# using custom color
colors  = ['skyblue','peachpuff','lightgreen','lightsalmon','lightcyan',
         'bisque','mistyrose','lightgray','wheat','lightblue',
         'lavender','linen','honeydew','cornsilk','lavender',
         'seashell','lightsteelblue','papayawhip','aliceblue','ghostwhite']

fig, ax = plt.subplots(figsize=(10,8))
ax.pie(sizes,explode=explode,labels=labels,
       autopct='%1.1f%%', colors=colors,
      shadow=True,startangle=180, textprops={'fontsize':15})

# Equal aspect ratio ensures that pie is drawn as a circle 
ax.axis('equal')
# setting title fontsize and padding between title and figure
plt.title('Percentage of top 20 tags based on count',
          fontsize=20,y=1.05)
plt.tight_layout()
plt.show()

From above pie-chart we can see that 19.3% of total which is 3785 times machine-learning tag was used in the question from 2019 to 2020(until now). People used this tag to ask question related to machine learning in the StackExchange Data Science platform.

In [45]:

labels = top_20_viewed_tags.index.values
values = top_20_viewed_tags['ViewCount']

fig = go.Figure(data=[go.Pie(labels=labels, values=values, 
                             textinfo='label+percent',
                             insidetextorientation='radial'
                            )])

fig.update_layout(showlegend=False,title='Top 20 viewed Data Science Tags')
fig.show()

In [61]:

# same for the pie-charts

#Pie chart
labels = top_20_viewed_tags.index.values
sizes = top_20_viewed_tags['ViewCount']

# only "explode" the first slice (i.e. value with highest count)
explode = (0.1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0)

# using custom color
colors  = ['skyblue','peachpuff','lightgreen','lightsalmon','lightcyan',
         'bisque','mistyrose','lightgray','wheat','lightblue',
         'lavender','linen','honeydew','cornsilk','lavender',
         'seashell','lightsteelblue','papayawhip','aliceblue','ghostwhite']

fig, ax = plt.subplots(figsize=(10,8))
ax.pie(sizes,explode=explode,labels=labels,
       autopct='%1.1f%%', colors=colors,
      shadow=True,startangle=180, textprops={'fontsize':15})

# Equal aspect ratio ensures that pie is drawn as a circle 
ax.axis('equal')
# setting title fontsize and padding between title and figure
plt.title('Percentage of top 20 tags based on view count',
          fontsize=20,y=1.05)
plt.tight_layout()
plt.show()

Similarly, above pie-chart shows that 19.9% of total in which python was tagged in the question had more than 1M viewers(not sure if StackExchange uses clicks as the viewCount(don't thing this is true) or uses user IP(Internet Protocol) address as the count but there are some assumptions here). These are the things you need to consider before setting proxy as actual parameter to determine our conclusion. For now, we considering what data has shown using proxy.

Trends over the period in the Tags¶

In this section, we'll see trends in the tags used in 2019 and 2020 by ploting the line graph for that tag.

In [36]:

import matplotlib.dates as mdates
import re


# function to return dataframe
def tag_df(df,tag_pattern):
    """
    Function to return filter dataFrame containing the specific tag
    
    Args:
        df: main dataFrame
        tag:pattern to filter the dataFrame
        
    Returns:
        Filtered DataFrame
    """
    
    filter_df = df[df['Tags'].apply(lambda tag:
                    True if re.search(tag_pattern,tag) else False)]

    return filter_df


# function to return df containing max and min value of numerical
# features
def max_min_df(df,tag_pattern,feature):
    """
    Function that returns df containing max and min value of
    numerical variables for given dataFrame and features
    
    Args:
        df: dataFrame
        tag_pattern: pattern to match for the exact tag
        feature: column in the dataFrame to filter the dataFrame
        
    Returns:
        dataFrame containing max and min values for given feature
    """
    
    # first filter the dataFrame based on the tag
    filter_df = tag_df(df,tag_pattern)
    
    # using boolean masking for filtering max and min values
    boolean_mask = (filter_df[feature] ==filter_df[feature].max()) | \
                    (filter_df[feature] ==filter_df[feature].min())
    
    return filter_df[boolean_mask]

# let's create a function which takes dataFrame and tag and returns line 
# graph with Date and ViewCount for that tag
def line_graph_tag(df,pattern,feature,interval=8):
    """
    Function plots a line graph for a given tag
    
    Args:
        df: dataFrame
        tag: target tag name to plot the graph
        feature: numerical feature you want to plot against date
        interval: interval for date, you want the data to be considered
        
    Returns:
        displays a line graph
    """
    
    # checking if the feature is numerical 
    if pd.api.types.is_numeric_dtype(df[feature][:1]):
        #print('printing graph')
        #print(type(pattern))
        
        t_df = tag_df(df,pattern)
        
        # setting the date as index
        t_df.set_index('CreationDate',inplace=True)
        
        #plot 
        fig, ax = plt.subplots(figsize=(15,7))
        t_df[feature].plot(ax=ax,color='cornflowerblue')
        
        # set ticks according to interval
        ax.xaxis.set_major_locator(mdates.WeekdayLocator(byweekday=mdates.MO,
                                                        interval=interval))
        
        # set major ticks format
        ax.xaxis.set_major_formatter(mdates.DateFormatter('%b %d'))
        
        tag = '{} tag questions trends'.format(pattern[2:-2])
        
        # set color
        ax.set_facecolor('whitesmoke')
        plt.title(tag,fontsize=25)
        plt.xticks(fontsize=15)
        plt.yticks(fontsize=15)
        plt.ylabel(feature,fontsize=15)
        
    else:
        print('function supports only numerical variabels')

In [57]:

# value counts for PostTypeId
df['PostTypeId'].value_counts()

Out[57]:

1    12755
2    12046
5       94
4       94
6        8
Name: PostTypeId, dtype: int64

When you run following sql command in StackExchange public database, we can see there are eight types of Posts and PostId equals 1 represents Question as we looking for most popular questions asked related to Data Science.

SELECT * FROM PostTypes;

In [10]:

# let's filter the data with post id=1
questions_df = df[df['PostTypeId'] == 1]
questions_df.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 12755 entries, 1 to 24995
Data columns (total 8 columns):
Id               12755 non-null int64
PostTypeId       12755 non-null int64
CreationDate     12755 non-null datetime64[ns]
Score            12755 non-null int64
ViewCount        12755 non-null int32
Tags             12755 non-null object
AnswerCount      12755 non-null int32
FavoriteCount    12755 non-null int32
dtypes: datetime64[ns](1), int32(3), int64(3), object(1)
memory usage: 747.4+ KB

Let's look at the deep-learning tag for now and see the its trend.

In [35]:

# trying with categorical value, which returns msg telling it only 
# supports numerical values
feature = 'Tags'
pattern = r'\bdeep-learning\b'

line_graph_tag(df_clean,pattern,feature)

function supports only numerical variabels

In [17]:

feature = 'ViewCount'
pattern = r'\bdeep-learning\b'

line_graph_tag(questions_df,pattern,feature)

Let's explore the trends of deep-learning tag in recent two months

In [13]:

import datetime as dt

feature = 'ViewCount'
pattern = r'\bdeep-learning\b'

# comparing the trends of question asked in deep-learning in recent two months
# i.e from june onwards
dl_2020_2m = questions_df[questions_df['CreationDate'] \
                            > dt.datetime(2020,6,1)]

# plot the graph with the help of the function
line_graph_tag(dl_2020_2m,pattern,feature,interval=1)

In [11]:

feature = 'ViewCount'
pattern = r'\bpython\b'

line_graph_tag(questions_df,pattern,feature)

In [30]:

# feature and pattern that you like to search for
feature = 'ViewCount'
pattern = r'\bpython\b'

# comparing the trends of question asked in python in recent two months
# i.e from june onwards
p_2020_2m = questions_df[questions_df['CreationDate'] \
                            > dt.datetime(2020,6,1)]

# plot the graph with the help of the function
line_graph_tag(p_2020_2m,pattern,feature,interval=1)

In [39]:

feature = 'ViewCount'
pattern = r'\br\b'

line_graph_tag(questions_df,pattern,feature)

In [40]:

# feature and pattern that you like to search for
feature = 'ViewCount'
pattern = r'\br\b'

# comparing the trends of question asked in R in recent two months
# i.e from june onwards
r_2020_2m = questions_df[questions_df['CreationDate'] \
                            > dt.datetime(2020,6,1)]

# plot the graph with the help of the function
line_graph_tag(r_2020_2m,pattern,feature,interval=1)

Let's find out the highest & lowest view count of the tags with respective month. For now, let's take python and r tag so that we can perform analysis regarding their popularity.

In [37]:

tag_pattern = r'\bpython\b'

# looking for max and min view count of tag python
max_min_df(questions_df,tag_pattern,'ViewCount')

Out[37]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
4653	56469	1	2019-07-27 00:27:57	0	4	python,deep-learning,tensorflow,object-detection	0	0
13026	74751	1	2020-05-24 14:01:09	0	4	python,time-series,vector-space-models,statsmo...	0	0
14785	45314	1	2019-02-09 18:24:50	6	88393	python,pandas	3	3

We can see from above that there is one Highest view count of python tag is 88393 in Feb,2019 whereas there are two lowest view count with 4 times in May,2019 & July 2019.

Now, What about the score features. Let's have a look.

In [38]:

#using the same tag_pattern as above cell and Score instead of ViewCount
max_min_df(questions_df,tag_pattern,'Score')

Out[38]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
2399	60890	1	2019-09-27 13:52:34	-2	33	python,visualization,matplotlib	1	1
9923	73977	1	2020-05-11 15:52:13	-2	24	machine-learning,python,scikit-learn,regressio...	1	1
12921	57721	1	2019-08-17 12:31:32	-2	32	machine-learning,python,data-cleaning,outlier	1	0
16888	58594	1	2019-09-03 09:48:16	-2	56	machine-learning,python,dataset	1	0
20639	60067	1	2019-09-11 19:36:52	-2	38	python,pandas	1	0
22902	76482	1	2020-06-22 17:55:40	-2	19	python,data,training,forecasting	1	0
24563	76824	1	2020-06-29 03:59:04	24	5955	python,r,statistics,data-analysis	7	7
24995	69683	1	2020-03-14 12:27:09	-2	27	machine-learning,python,logistic-regression	1	0

Interesting, didn't expected Score value to be in negative. What I found out here after googling: negative values are due to down votes on the post, makes sense. That fulfills my curiosity.

Let's move on to r tag.

In [41]:

tag_pattern = r'\br\b'

# looking for max and min view count of tag python
max_min_df(questions_df,tag_pattern,'ViewCount')

Out[41]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
17548	46050	1	2019-02-22 17:32:52	5	13556	r,time-series,forecast,data-analysis	4	4
20637	60064	1	2019-09-11 19:16:35	0	4	r	0	0

The highest view count was 13556 on Feb 2019 and the lowest count was 4 on Aug 2019.

In [42]:

#using the same tag_pattern as above cell and Score instead of ViewCount
max_min_df(questions_df,tag_pattern,'Score')

Out[42]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
219	52511	1	2019-05-24 06:46:23	-1	36	r	1	0
2064	68437	1	2020-02-21 05:32:59	-1	23	machine-learning,r,linear-regression	1	0
6878	53843	1	2019-06-15 10:05:22	-1	51	r,regression	1	1
7515	47757	1	2019-03-21 19:47:10	-1	52	r,data-mining,dataset,data,data-cleaning	2	0
9707	57424	1	2019-08-12 07:33:28	-1	35	machine-learning,r,machine-learning-model	1	0
10578	54452	1	2019-06-25 11:51:11	-1	81	r,programming	2	1
12925	57725	1	2019-08-17 18:31:00	-1	162	python,r,data-science-model,data-analysis,mark...	1	0
13488	55166	1	2019-07-06 03:31:01	-1	53	r	1	0
18713	66949	1	2020-01-23 20:23:44	-1	26	machine-learning,r	1	0
21683	72433	1	2020-04-16 12:54:30	-1	16	r,visualization,rstudio	1	0
23307	52170	1	2019-05-18 19:27:20	-1	29	machine-learning,r,rstudio	2	0
24563	76824	1	2020-06-29 03:59:04	24	5955	python,r,statistics,data-analysis	7	7

Minimium score of -1(down votes) for 11 post and maximun score of 24(upvotes) can be seen in the above table.

Common and Uncommon tags between both top 20 tags¶

Let's check what is common and what is uncommon in between most used and most viewed tags. For this purpose, we'll be using pandas merge function.

In [38]:

in_used = pd.merge(top_20_used_tags,top_20_viewed_tags,how='left',
                  left_index=True,right_index=True)
in_used

Out[38]:

	Count	ViewCount
machine-learning	3785	865175.0
python	2594	1300942.0
deep-learning	1705	496582.0
neural-network	1488	416241.0
keras	1291	595612.0
classification	1004	233819.0
tensorflow	869	293228.0
scikit-learn	801	309171.0
nlp	748	179552.0
cnn	667	166664.0
time-series	646	142171.0
lstm	544	157203.0
regression	532	107170.0
pandas	525	566170.0
dataset	505	92937.0
clustering	422	NaN
r	398	88902.0
predictive-modeling	377	NaN
statistics	368	NaN
machine-learning-model	320	NaN

In [39]:

in_view = pd.merge(top_20_used_tags,top_20_viewed_tags,how='right',
                  left_index=True,right_index=True)
in_view

Out[39]:

	Count	ViewCount
python	2594.0	1300942
machine-learning	3785.0	865175
keras	1291.0	595612
pandas	525.0	566170
deep-learning	1705.0	496582
neural-network	1488.0	416241
scikit-learn	801.0	309171
tensorflow	869.0	293228
classification	1004.0	233819
dataframe	NaN	214866
nlp	748.0	179552
cnn	667.0	166664
lstm	544.0	157203
time-series	646.0	142171
numpy	NaN	122505
regression	532.0	107170
pytorch	NaN	94009
dataset	505.0	92937
csv	NaN	90857
r	398.0	88902

Above, NaN(Not a Number) after merge shows tags not included in top 20 in both the cases.

Tags present in top_20_used_tags and not present in top_20_viewed_tags are:

• clustering
• predictive-modeling
• statistics
• machine-learning-model

And in other way around, tags not present are:

• dataframe
• numpy
• pytorch
• csv

So, there were couple of tags which were more used and some were viewed more.

Insight from our analysis:¶

In this project, we analyzed the data from StackExchange Data Science public database to find the popular tags in Data Science. If you're familiar with the data science realm, you can see that quarter of the tags used are from Deep Learning or AI. Python libraries like Pandas, NumPy, and Scikit-learn are in the top 20 tags used. This gives an idea of Python as a majority's choice to solve data science problems.

We can see the fluctuation in the tags trends over the period of 2019 and 2020(till July 12) from the line graphs above. But if you consider the month to month comparison, the year 2020 has a lower number of counts in the tags used than 2019.

Relationship between multiple tags

We can see that there are multiple tags for a single question, so is there any relationship between tags. That's a topic for discussion which we're hoping to address in later sections. For now, just get an idea of mutual tags available.

In [41]:

import re

# looking for multiple tags pair: 'r' and 'pandas'
temp_df[temp_df["Tags"].apply(
    lambda tags: True if re.search(r'\br\b', tags) # regex to look for 'r'
    and "pandas" in tags else False)
]

Out[41]:

	Id	PostTypeId	CreationDate	Score	ViewCount	Tags	AnswerCount	FavoriteCount
8562	49148	1	2019-04-11 19:41:39	1	133	r,data-mining,pandas,matlab,databases	3	0
10528	60074	1	2019-09-11 20:35:17	0	94	r,pandas,dplyr	0	0

Questions to look for in order to get idea about the tags relationship

• Is there any pair of tags(considering tag pair if it appears in more than 10(threshold, you can define whatever you like),considering it doesn't appear coincidently)?
• Is there any relationship between multiple tags(you can consider a mutual tag between two tags)?

For future purpose, we can consider association rules or clustering techniques to find out group of tags that come together similar to Market Basket Analysis done in retail industry(what items customer buy together: bread & butter).

Follow up with:

• Apply association rules(mining) to determine tags that come together

Let's try Association mining for this time.¶

Let's start preparing data for association mining.

Apriori Algorithm¶

Note: apyori library only supports python version between 3.3-3.5, so we're using efficient_apriori library as it supports python 3.6 and above.

In [33]:

from efficient_apriori import apriori

Data processing¶

In order to feed the data to the algorithm, we've to convert dataFrame into list of lists or list of tuples. We're lucky, as we've Tags values in the form of string seperated by comma. So, using string split method will produce list. After that, just append the resulting list to list, that gives us our list of lists.

In [34]:

print('Total number of data points = {}'.format(len(df_clean)))

Total number of data points = 24997

In [35]:

# our final list of lists
tag_records = []

for i in range(0,24997):
    # pass for nan
    if type(df_clean.loc[i,'Tags']) != float:
        tag_records.append(df_clean.loc[i,'Tags'].split(','))

In [36]:

# our list of lists
tag_records[:10]

Out[36]:

[['lstm', 'loss-function', 'epochs'],
 ['deep-learning', 'cnn'],
 ['time-series', 'clustering', 'unsupervised-learning', 'anomaly-detection'],
 ['machine-learning', 'python', 'dataset', 'training', 'object-detection'],
 ['lstm', 'code'],
 ['lstm'],
 ['reinforcement-learning'],
 ['classification', 'regression', 'feature-extraction', 'feature-engineering'],
 ['pytorch'],
 ['scikit-learn', 'decision-trees']]

Now, let's feed the data to the apriori algorithm to analyze the results. Before that, let's get idea of the major components of apriori algorithms:

• Support: ratio of number of times the item appears in the total transaction.
• Confidence: ratio of items A and B appears together in total number of transaction of item A.
• Left: Likelihood of item B appears given item A.

For example, we're taking min_support=0.002 to make sure there are at least 50 data points containing the tags out of 24997. To make sure 10 out of 50 data points have items together, we set min_confidence=0.2.

In [37]:

itemsets,rules = apriori(tag_records,min_support=0.002,
                        min_confidence=0.2)

In [38]:

print('Total number of rules produce by algorithm is {}'.format(len(rules)))

Total number of rules produce by algorithm is 329

In [40]:

# Print out every rule with 2 items on the left hand side,
# 1 item on the right hand side, sorted by lift
rules_rhs = filter(lambda rule: len(rule.lhs) >= 2 and \
                   len(rule.rhs) == 1, rules)
for rule in sorted(rules_rhs, key=lambda rule: rule.lift,
                  reverse=True)[:31]:
    print(rule)  # Prints the rule and its confidence, support, lift, ...

{dataframe, python} -> {pandas} (conf: 0.833, supp: 0.005, lift: 20.246, conv: 5.753)
{forecasting, machine-learning} -> {time-series} (conf: 0.732, supp: 0.002, lift: 14.447, conv: 3.538)
{keras, rnn} -> {lstm} (conf: 0.591, supp: 0.002, lift: 13.855, conv: 2.340)
{keras, time-series} -> {lstm} (conf: 0.552, supp: 0.003, lift: 12.948, conv: 2.138)
{feature-selection, machine-learning} -> {feature-engineering} (conf: 0.224, supp: 0.002, lift: 12.416, conv: 1.265)
{deep-learning, lstm} -> {rnn} (conf: 0.218, supp: 0.002, lift: 12.344, conv: 1.256)
{feature-engineering, machine-learning} -> {feature-selection} (conf: 0.280, supp: 0.002, lift: 11.353, conv: 1.355)
{deep-learning, rnn} -> {lstm} (conf: 0.482, supp: 0.002, lift: 11.305, conv: 1.849)
{numpy, python} -> {pandas} (conf: 0.343, supp: 0.003, lift: 8.323, conv: 1.459)
{lstm, python} -> {keras} (conf: 0.598, supp: 0.004, lift: 5.904, conv: 2.233)
{deep-learning, image-classification} -> {cnn} (conf: 0.302, supp: 0.002, lift: 5.777, conv: 1.358)
{image-classification, machine-learning} -> {cnn} (conf: 0.301, supp: 0.002, lift: 5.755, conv: 1.356)
{cnn, tensorflow} -> {keras} (conf: 0.575, supp: 0.004, lift: 5.681, conv: 2.115)
{lstm, tensorflow} -> {keras} (conf: 0.545, supp: 0.002, lift: 5.389, conv: 1.977)
{neural-network, tensorflow} -> {keras} (conf: 0.519, supp: 0.006, lift: 5.128, conv: 1.869)
{python, tensorflow} -> {keras} (conf: 0.517, supp: 0.009, lift: 5.112, conv: 1.862)
{cnn, python} -> {keras} (conf: 0.514, supp: 0.003, lift: 5.081, conv: 1.850)
{python, regression} -> {scikit-learn} (conf: 0.300, supp: 0.002, lift: 4.777, conv: 1.339)
{machine-learning, tensorflow} -> {keras} (conf: 0.482, supp: 0.006, lift: 4.766, conv: 1.736)
{keras, python} -> {tensorflow} (conf: 0.323, supp: 0.009, lift: 4.746, conv: 1.377)
{deep-learning, tensorflow} -> {keras} (conf: 0.466, supp: 0.008, lift: 4.607, conv: 1.684)
{keras, machine-learning} -> {tensorflow} (conf: 0.304, supp: 0.006, lift: 4.458, conv: 1.338)
{cnn, keras} -> {tensorflow} (conf: 0.301, supp: 0.004, lift: 4.413, conv: 1.332)
{deep-learning, keras} -> {tensorflow} (conf: 0.283, supp: 0.008, lift: 4.151, conv: 1.299)
{cnn, deep-learning, machine-learning} -> {neural-network} (conf: 0.474, supp: 0.002, lift: 4.060, conv: 1.678)
{keras, lstm} -> {time-series} (conf: 0.201, supp: 0.003, lift: 3.970, conv: 1.188)
{keras, neural-network} -> {tensorflow} (conf: 0.268, supp: 0.006, lift: 3.933, conv: 1.273)
{neural-network, python} -> {keras} (conf: 0.388, supp: 0.006, lift: 3.833, conv: 1.469)
{deep-learning, python} -> {keras} (conf: 0.386, supp: 0.007, lift: 3.814, conv: 1.464)
{numpy, pandas} -> {python} (conf: 0.740, supp: 0.003, lift: 3.639, conv: 3.064)
{cnn, machine-learning} -> {neural-network} (conf: 0.414, supp: 0.005, lift: 3.549, conv: 1.507)

Above, you can see the top 31 association rule sorted by lift value in descending order. Let's take the last one for a quick explanation. Taking the number of data points where both the tags cnn & machine-learning appeared and dividing by total gives support value=0.005. Confidence=0.414 explains that out of all the data points containing cnn & machine-learning tag, 41.4% of the data points also contain neural-network tag. Finally, lift=3.549 tells us that when cnn & machine-learning are used together there is likelihood of 3.549 for neural-network to appear with them.

Note: Here, data points used instead of explaining things with transaction (from Market Basket Analysis) cause it's more relevant here in case, people want to related back to it.

Final thoughts:¶

Techniques like apriori algorithms mostly are used in customer transaction data. It is used to identify the items that are bought together by the customer. Insight gain from these are used later in promotion, placement of the items in the store, and many more.

Here, we used this technique to identify the relationship between multiple tags used in the post at StackExchange forum. Total rules produced by the algorithm are 329 out of 148 different tags used. This shows that there are many-to-many relationships between tags.